Image processing device

ABSTRACT

Provided is an image processing device for sorting frames included in one moving image into different channels, the frames included in the moving image being a mixture of frames belonging to the different channels. The image processing device includes an automatic sorting unit and a re-sorting unit. The automatic sorting unit calculates a similarity degree of the plurality of frames included in the moving image by performing image processing on the frames, and sorts the plurality of frames into the plurality of channels based on the similarity degree. The re-sorting unit accepts, from a user, an association operation of individually associating at least one non-sorted frame, which is not automatically sorted by the automatic sorting unit, with any of the plurality of channels, and individually sorts the non-sorted frame to any of the plurality of channels based on content of the association operation.

TECHNICAL FIELD

The present invention relates to an image processing device and an imageprocessing program, and particularly relates to an image processingdevice and image processing program that are configured to sort framesincluded in one moving image into different channels, the framesincluded in the moving image being a mixture of frames that belong tothe different channels.

BACKGROUND ART

An intermittent recording method, called “time-lapse”, is known as amoving image recording method. This is a method for recording an imageat a lower frame rate than usual, that is, in a time-lapse manner, andenables recording for a longer time period than usual to the samerecording medium. Security cameras often employ the time-lapse method,and furthermore a video picture from another security camera may beinserted into a time period gap that is generated by lowering the framerate. In this case, a moving image in which video pictures from aplurality of cameras, that is, video pictures of a plurality of channelsare switched alternately along one time-line, is recorded.

With respect to the moving image in which video pictures of a pluralityof channels are recorded together on one time-line, video pictures fromeach camera, that is, each channel, can be displayed on a monitor withdedicated equipment. In the dedicated equipment of Patent Literature 1,channel number data is added to the frames of video pictures of aplurality of channels, and at the time of reproduction, the dedicatedequipment can select only the video pictures of a desired channel fromamong the video pictures of the plurality of channels by referencing thechannel number data.

CITATION LIST Patent Literature

Patent Literature 1: JP 2002-319210A

SUMMARY OF INVENTION Technical Problem

However, when reproducing video pictures in which video pictures ofmultiple channels are mixed with each other, using general-purposereproduction equipment, it is impossible to view video pictures for eachchannel. The specifications for recognizing the channel of each framediffers between manufacturers or device types, and thus general-purposereproduction equipment cannot support these specifications.

It is an object of the present invention to provide an image processingdevice and an image processing program that enable, when one movingimage includes a mixture of frames that belong to different channels,reproduction for each channel regardless of specifications by which themoving image is recorded.

Solution to Problem

According to a first aspect of the present invention, an imageprocessing device is configured to sort frames included in one movingimage into different channels, the frames included in the moving imagebeing a mixture of frames belonging to the different channels, andincludes an automatic sorting unit and a re-sorting unit. The automaticsorting unit calculates a similarity degree of the plurality of framesincluded in the moving image by performing image processing on theframes, and sorts the plurality of frames into the plurality of channelsbased on the similarity degree. The re-sorting unit accepts, from auser, an association operation of individually associating at least onenon-sorted frame, which is not automatically sorted by the automaticsorting unit, with any of the plurality of channels, and individuallysorts the non-sorted frame to any of the plurality of channels based oncontent of the association operation.

Here is provided a function for sorting, when one moving image includesa mixture of frames belonging to different channels, the frames by thechannels. Specifically, first, a degree of similarity (similaritydegree) between the plurality of frames included in the moving image iscalculated by performing image processing on the frames, and theplurality of frames are automatically sorted by the plurality ofchannels based on the similarity degree. However, if video pictures,even belonging to different channels, are of a scene in which, forexample, both channels have a similar background or the like iscaptured, the accuracy of automatic sorting is not necessarily perfect.Accordingly, here, re-sorting based on a user operation is performedafter the automatic sorting. Specifically, the user can perform theoperation of individually associating the non-sorted frame with any ofthe plurality of channels after the automatic sorting, and re-sorting isperformed based on this operation. According to the function, regardlessof the specifications of the moving image, it is possible to easily andprecisely separate the moving image that includes a mixture of framesbelonging to different channels into moving images for the respectivechannels. That is, when one moving image includes a mixture of framesbelonging to different channels, it is possible to perform reproductionfor each channel, regardless of under which specifications the movingimage is recorded.

According to a second aspect of the present invention, the imageprocessing device according to the first aspect is such that there-sorting unit displays a re-sorting screen for displaying thenon-sorted frame, and a plurality of channel objects, which are objectsrespectively corresponding to the plurality of channels. The associationoperation is an operation for designating, on the re-sorting screen, anyof the plurality of channel objects to the non-sorted frame. Note thatthe re-sorting screen may be constituted by one window or a plurality ofwindows. Furthermore, displaying a frame includes displaying a thumbnailimage of the frame.

Here, the re-sorting screen is provided as a user interface forperforming re-sorting after the automatic sorting. This re-sortingscreen displays a non-sorted frame, and a plurality of channel objectsrespectively corresponding to the plurality of channels, and accepts,from the user, an operation for designating any of the channel objectsto the non-sorted frame. Accordingly, the user can associate and sortthe non-sorted frame to a specific channel with an intuitive operation.

According to a third aspect of the present invention, the imageprocessing device according to the second aspect is such that eachchannel object displays a representative image that represents a framegroup belonging to a corresponding channel.

Here, the channel object displays a representative image that representsthe channel corresponding to the channel object. Accordingly, the usercan compare the non-sorted frame with the representative image with hisor her eyes, and then can designate the channel object to the non-sortedframe. The user can thus view the re-sorting screen, and thereby easilydetermine the channel to which the non-sorted frame is to be sorted.

According to a fourth aspect of the present invention, the imageprocessing device according to the second or third aspect is such thatthe re-sorting screen displays the at least one non-sorted frame in alist.

Here, on the re-sorting screen, the at least one non-sorted frame isdisplayed in a list. Accordingly, the user can check such frames at thesame time.

According to a fifth aspect of the present invention, the imageprocessing device according to the fourth aspect is such that there-sorting screen further displays a non-sorted object, which is anobject corresponding to a sort of putting the at least one non-sortedframe together. The re-sorting unit accepts, from the user, a selectionoperation of selecting one object from among an object group includingthe plurality of channel objects and the non-sorted object, and displaysa frame group in a list in a predetermined list display area on there-sorting screen. The flame group belongs to a channel or sort thatcorresponds to the object currently selected by the selection operation.

Here, by selecting one object from among the non-sorted object and theplurality of channel objects, the user can selectively display, in thepredetermined list display area, the non-sorted frame group or the framegroup belonging to any of the channels. Accordingly, it is possible tocheck non-sorted and sorted frames while advantageously using a limitedspace within the re-sorting screen.

According to a sixth aspect of the present invention, the imageprocessing device according to any one of the second to fourth aspectsis such that the re-sorting unit accepts, from the user, a selectionoperation of selecting one object from among an object group includingthe plurality of channel objects, and displays a frame group in a listin a predetermined list display area on the re-sorting screen. The framegroup belongs to a channel that corresponds to the channel objectcurrently selected by the selection operation.

Here, by selecting one object from among the plurality of channelobjects, the user can selectively display, in the predetermined listdisplay area, the frame group belonging to any channel. Accordingly, itis possible to check sorted frames while advantageously using a limitedspace within the re-sorting screen.

According to a seventh aspect of the present invention, the imageprocessing device according to the fifth or sixth aspect is such that,when a frame group belonging to any of the plurality of channels isdisplayed in a list in the list display area, the re-sorting unitaccepts, from the user, an operation for individually associating aframe that is included in the frame group displayed in the list with anyof the plurality of channels, and sorts the frame as per the operationto the channel as per the operation.

Here, the frame that is displayed in the list display area and wassorted to any channel can be moved to a different channel. Specifically,the user can perform an operation for individually associating a framethat was sorted to any channel by the automatic sorting with any ofdiffering channels, and re-sorting is performed based on the operation.Accordingly, if a frame is sorted to a wrong channel by the automaticsorting, the frame can be corrected.

According to an eighth aspect of the present invention, the imageprocessing device according to any one of the second to seventh aspectsis such that the re-sorting unit accepts, from the user, a selectionoperation of selecting one object from among an object group includingthe plurality of channel objects, and reproduces a frame group as amoving image in a predetermined reproduction area on the re-sortingscreen. The frame group belongs to a channel that corresponds to theobject currently selected by the selection operation,

Here, it is possible to reproduce, on the re-sorting screen, a framegroup sorted to each channel as a moving image. Accordingly, if a frameis wrongly sorted to a channel, it is possible to find this frameeasily.

According to a ninth aspect of the present invention, the imageprocessing device according to any one of the second to eighth aspectsis such that the association operation includes an operation of draggingand dropping the non-sorted frame onto any of the plurality of channelobjects, or dragging and dropping any of the plurality of channelobjects onto the non-sorted frame, on the re-sorting screen.

Here, the user can further intuitively associate and sort a non-sortedframe to a specific channel.

According to a tenth aspect of the present invention, the imageprocessing device according to any one of the second to ninth aspects issuch that the re-sorting screen displays a group of frames belonging toeach channel in order of time series in a list. Note that the framegroup belonging to another channel is not necessarily displayed at thesame time on the re-sorting screen, and may be displayed in, forexample, a switched manner.

Here, a group of frames belonging to each channel are arranged in orderof time series. Accordingly, the user can easily understand a change inthe video picture (movement of a subject) of each channel.

According to an eleventh aspect of the present invention, the imageprocessing device according to any one of the first to tenth aspects issuch that the re-sorting unit displays information indicating anoperation key that corresponds to each of the plurality of channels. Theassociation operation includes operation of the operation key.

Here, the user can associate and sort a non-sorted frame to a specificchannel with an operation using the operation key of a keyboard or thelike.

According to a twelfth aspect of the present invention, the imageprocessing device according to any one of the first to eleventh aspectsis such that the re-sorting unit recalculates the frame rate each timethe non-sorted frame is sorted to any of the plurality of channels.

The number of frames belonging to each channel changes each time anon-sorted frame is newly sorted to the channel. Here, since the framerate of the channel is recalculated in accordance with the change, thechannel can maintain a correct frame rate.

According to a thirteenth aspect of the present invention, the imageprocessing device according to any one of the first to twelfth aspectsis such that the re-sorting unit accepts, from the user, an instructionto combine two or more channels included in the plurality of channels,and combines the two or more channels in response to the instruction forcombination.

Here, it is possible to combine two or more channels created by theautomatic sorting. Accordingly, even if one channel is separated intotwo or more channels by mistake with the automatic sorting, it will bepossible to easily correct the mistake.

According to a fourteenth aspect of the present invention, the imageprocessing device according to any one of the first to thirteenthaspects is such that the re-sorting unit accepts, from the user, aninstruction to create a channel other than the plurality of channelsautomatically created by the automatic sorting unit, and newly createsthe channel in response to the instruction for creation.

Here, it is possible to create a new channel at the time of re-sortingafter the automatic sorting. Accordingly, even if there is a channelthat was not detected by automatic sorting, it will be possible toeasily correct this.

According to a fifteenth aspect of the present invention, an imageprocessing program is for sorting frames included in one moving imageinto different channels, the frames included in the moving image being amixture of frames belonging to the different channels the imageprocessing program causing a computer to execute the steps of:calculating a similarity degree of the plurality of frames included inthe moving image by performing image processing on the frames; sortingthe plurality of frames into the plurality of channels based on thesimilarity degree; accepting, from a user, an association operation ofindividually associating at least one non-sorted frame, which is notautomatically sorted, with any of the plurality of channels; andindividually sorting the non-sorted frame to any of the plurality ofchannels based on content of the association operation.

Advantageous Effects of Invention

According to the present invention, it is possible to easily andprecisely separate a moving image that includes a mixture of framesbelonging to different channels into moving images for the respectivechannels, regardless of the specifications of the moving image. That is,when one moving image includes a mixture of frames belonging todifferent channels, reproduction for each channel is possible regardlessof specifications by which the moving image was recorded.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an image processing deviceaccording to a first embodiment of the present invention.

FIG. 2 is a diagram illustrating a basic screen before image data isloaded.

FIG. 3 is a diagram illustrating the basic screen after the image datais loaded.

FIG. 4 is a diagram illustrating a still image group that belongs to onetime-line.

FIG. 5 is a diagram illustrating an area setting window.

FIG. 6 is a diagram illustrating a re-sorting window.

FIG. 7 is another diagram illustrating the re-sorting window.

FIG. 8 is a diagram illustrating a check window.

FIG. 9 is a flowchart illustrating a flow of automatic sortingprocessing according to the first embodiment of the present invention.

FIG. 10 is a schematic diagram illustrating a method to calculate asimilarity degree according to the first embodiment of the presentinvention.

FIG. 11 is a flowchart illustrating a flow of automatic sortingprocessing according to a second embodiment of the present invention.

FIG. 12 is a schematic diagram illustrating a method to calculate asimilarity degree according to the second embodiment of the presentinvention.

FIG. 13 is a flowchart illustrating a flow of automatic sortingprocessing according to a third embodiment of the present invention.

FIG. 14 is a schematic diagram illustrating a method to calculate asimilarity degree according to the third embodiment of the presentinvention.

FIG. 15 is a schematic diagram illustrating another method to calculatea similarity degree according to the third embodiment of the presentinvention.

FIG. 16 is a schematic diagram illustrating yet another method tocalculate a similarity degree according to the third embodiment of thepresent invention.

FIG. 17 is a flowchart illustrating a flow of processing for evaluatinga color trend according to the third embodiment of the presentinvention.

FIG. 18 is a diagram illustrating impression defining information.

DESCRIPTION OF EMBODIMENTS

Hereinafter, an image processing device and an image processing programaccording to some embodiments of the present invention will be describedwith reference to the drawings.

1. First Embodiment

<1-1. Summary of Image Processing Device>

An image processing device 1 of FIG. 1 shows an embodiment of the imageprocessing device according to the present invention. The imageprocessing device 1 is a general-purpose personal computer. An imageprocessing program 2, which is an embodiment of an image processingprogram according to the present invention, is provided and installed inthe image processing device 1, for example, from a computer readablestorage medium 60 such as a CD-ROM, DVD-ROM, or a USB memory that storesthe image processing program 2. The image processing program 2 isapplication software for assisting in image processing to be performedon a moving image and a still image. The image processing program 2causes the image processing device 1 to execute steps included in theoperations that will be described later.

The image processing device 1 includes a display 10, an input unit 20, astorage unit 30, and a control unit 40. These units 10 to 40 areconnected to each other via, for example, a bus line or cable 5, and cansuitably perform communication with each other. The display 10 isconstituted by a liquid crystal display or the like, and displays, to auser, a screen or the like that will be described later. The input unit20 is constituted by a mouse, a keyboard, and the like, and accepts auser operation on the image processing device 1. The storage unit 30 isa nonvolatile storage area that is constituted by a hard disk, or thelike. The control unit 40 is constituted by a CPU, a ROM, a RAM, and thelike.

The image processing program 2 is stored in the storage unit 30. Thestorage unit 30 secures a software management region 50. The softwaremanagement region 50 is a region for use by the image processing program2. The software management region 50 secures an original image region51, a processed file region 52, and an impression defining region 53.The functions of the regions 51 to 53 will be described later.

By reading out and executing the image processing program 2 stored inthe storage unit 30, the control unit 40 virtually operates as anautomatic sorting unit 41 and a re-sorting unit 45. Furthermore, theautomatic sorting unit operates also as a setting unit 42, a calculationunit 43, and a determination unit 44. The operations of the units 41 to45 will be described later in detail.

<1-2. Details of Configuration and Operation of Image Processing Device>

Upon detecting that the user has performed a predetermined operation viathe input unit 20, the control unit 40 starts the image processingprogram 2. After the start-up of the image processing program 2, a basicscreen W1 (see FIG. 2) is displayed on the display 10. Note that displayof all elements such as screens, windows, and buttons displayed on thedisplay 10 is controlled by the control unit 40.

<1-2-1. Image Data Loading>

The basic screen W1 accepts, from the user, an instruction to load imagedata to the original image region 51. The image data loaded to theoriginal image region 51 is to be subjected to reproduction processing,image processing, and channel separation processing, which will bedescribed later. The control unit 40 loads image data from a still imagefile or a moving image file to the original image region 51. Note thatin the present specification, “still image file” refers to a data filein a still image format, and “moving image file” refers to a data filein a moving image format.

In the case of loading image data from a still image file, the useroperates the input unit 20 so as to designate one still image file orone folder. In the case of the former, the control unit 40 prompts theuser to input the address path of this still image file in the storageunit 30 and the file name thereof. In the case of the latter, thecontrol unit 40 prompts the user to input the address path of thisfolder in the storage unit 30 and the folder name thereof. Then, thecontrol unit 40 saves the designated still image file or all still imagefiles in the designated folder, as a still image file group, in theoriginal image region 51. Note that in the present specification,“group” does not necessarily include a plurality of elements but may beone element.

On the other hand, in the case of loading of image data from a movingimage file, the user operates the input unit 20 so as to input theaddress path of the one moving image file in the storage unit 30 and thefile name thereof. Upon detecting that the user has designated themoving image file, the control unit 40 displays a moving image loadwindow (not shown) on the basic screen W1 in a superimposed manner. Themoving image load window accepts, from the user, the selection of anarbitrary segment from among the entire segment on the time-line of thedesignated moving image file. Upon detecting that the user has selecteda specific segmental the input unit 20, the control unit 40 generates astill image file group that has a one-to-one correspondence with a framegroup included in the segment of the designated moving image file. Then,the control unit 40 saves the still image file group in the originalimage region 51. Accordingly, in the present embodiment, image data thatis to be subjected to reproduction processing and channel separationprocessing, which will be described later, is not of a moving image filebut of a still image file.

Note that even if the still image file group loaded to the originalimage region 51 is derived from a still image file instead of a movingimage file, the control unit 40 recognizes that the still image filegroup is sequenced along one time-line. The sequence is automaticallymade based on a file attribute or the like.

<1-2-2. Reproduction Processing>

After the still image file group is loaded to the original image region51, the control unit 40 displays display windows W2 (see FIG. 3) on thebasic screen W1 in a superimposed manner. The number of generateddisplay windows W2 corresponds to the number of time-lines of the stillimage file group loaded to the original image region 51.

First, one still image file (for example, the still image file of thefirst frame in the time-line) that is included in the still image filegroup loaded to the original image region 51 is displayed in a displaywindow W2. Then, as will be described later, the frame that is displayedin the display window W2 is switched upon an operation of the user.

The control unit 40 can reproduce, within a display window W2, the framegroup that belongs to the time-line corresponding to this display windowW2, as a moving image. Here, as shown in FIG. 3, a window selectionpull-down menu T1, a play button T2, a next frame button T3, a previousframe button T4, and a time-line bar T5 are arranged on the basic screenW1.

Even if a plurality of display windows W2 are present, there is oneactive display window W2. The window selection pull-down menu T1accepts, from the user, the selection of which display window W2 is toserve as the active display window W2. Hereinafter, the time-line thatcorresponds to the active display window W2 is referred to as “activetime-line”, and the frame group that belongs to the active time-line isreferred to as “active frame group”. Furthermore, the frame that iscurrently displayed in the active display window W2 is referred to as“active display frame”.

The play button T2 accepts an instruction to reproduce the active framegroup as a moving image from the user. Upon detecting that the user haspressed the play button T2 via the input unit 20, the control unit 40displays, within the active display window W2, the frames included inthe active frame group sequentially along the time-line in aframe-by-frame format. Note that the reproduction starts from the activedisplay frame at a point in time when the play button T2 is pressed.Furthermore, the play button T2 accepts an instruction to stop thereproduction from the user. Upon detecting that the user has pressed theplay button T2 via the input unit 20 during the reproduction, thecontrol unit 40 fixes the display of the active display window W2 to theactive frame at that point in time.

The next frame button T3 and the previous frame button T4 respectivelyaccept, from the user, instructions to switch the active display frameto the next frame and to the previous frame along the active time-line.

The time-line bar T5 is an object schematically showing the activetime-line. The time-line bar T5 is equally divided as per the number offrames included in the active frame groups in the direction in which thebar extends. The n-^(th) divided region from the left of the time-linebar T5 corresponds to the n-^(th) frame of the active time-line (n is anatural number).

As shown in FIG. 3, the time-line bar T5 displays a divided region A1that corresponds to a selected frame group and a divided region A2 thatcorresponds to a non-selected frame group in different modes. “Selectedframe group” refers to a frame group that corresponds to a currentlyselected segment of the active time-line. “Non-selected frame group”refers to a frame group that corresponds to a currently non-selectedsegment of the active time-line.

The time-line bar T5 accepts the selection of an arbitrary segment inthe active time-line from the user. In other words, by operating thedivided region on the time-line bar T5 via the input unit 20, the usercan select an arbitrary number of arbitrary frames from among the activeframe group. The control unit 40 recognizes the selected frame group asa target of image processing and channel separation processing, whichwill be described later. Note that, each time the user selects a dividedregion of the time-line bar T5, the active display frame is switched tothe frame that corresponds to the most recently selected divided region.

<1-2-3. Image Processing>

Hereinafter, the image processing that is to be performed on theselected frame group will be described. The control unit 40 can execute,with respect to the selected frame group, a plurality of imageprocessing modules such as noise removal, sharpness,brightness/contrast/chroma adjustment, image resolution, rotation,addition of characters/arrows/mosaics, and image averaging. The imageprocessing modules are incorporated in the image processing program 2.

The user can select arbitrary modules from the image processing modulesin an arbitrary order for an arbitrary number of times, by performing anoperation on the basic screen W1 via the input unit 20. Each time thecontrol unit 40 detects that the user has selected an image processingmodule, the control unit 40 executes the image processing module withrespect to the selected frame group at that point in time.

As the image processing module is sequentially executed with respect toa frame once, twice, trice, . . . , the frame is sequentially processedto a first-order frame, a second-order frame, a third-order frame, . . .The 0-order frame corresponds to the still image file saved in theoriginal image region 51. The (m+1)-order frame corresponds to the stillimage file obtained by executing the image processing module once withrespect to the still image file corresponding to the m-order frame (m isan integer of 0 or more). The control unit 40 sequentially generates thestill image files that correspond to the first-order frames onward, andsaves these still image files separately in the processed file region52.

FIG. 4 is a schematic diagram illustrating how the still image groupthat belongs to one time-line is managed by the image processing program2. In FIG. 4, the horizontal N axis denotes the order of frames on thetime-line, and the vertical M axis denotes the order of processing. Eachsquare of FIG. 4 that corresponds to the coordinates (n, m) in the N-Mspace denotes a still image Q (n, m). The still image Q (n, m) is them-order still image of the n-^(th) frame on the time-line (n is anatural number and m is an integer of 0 or more).

The control unit 40 manages, for each frame, the value of the currentlyselected coordinate m as a parameter m_(s). Immediately after the stillimage file group is loaded to the original image region 51, thecoordinate m_(s) has the default value 0. Then, the coordinate m_(s) ofthe frame is incremented by 1 each time the image processing module isexecuted. Furthermore, the user can freely change the coordinate m_(s)of a selected frame group by performing a predetermined operation viathe input unit 20. Note that executing an image processing module withrespect to a frame refers to executing the image processing module withrespect to the m_(s)-order still image of this frame. Accordingly,changing the coordinate m_(s) means changing the target with respect towhich the image processing module is to be executed. Displaying a framerefers to displaying the still image of the coordinate m_(s) of thisframe. Accordingly, changing the coordinate m_(s) also means changingthe target to be displayed in the active display window W2.

<1-3 . Channel Separation Processing>

Hereinafter, the channel separation processing that is implemented inthe image processing program 2 will be described. “Channel separationprocessing” refers to processing for separating a moving image thatincludes a mixture of video pictures belonging to different channelsinto moving images for the respective channels. That is, the movingimage to be subjected to the channel separation processing is typicallya moving image (hereinafter, referred to as “mixed time-lapse movingimage”) into which a plurality of moving images recorded using atime-lapse method are mixed on one time-line in a manner such that avideo picture of each moving image is embedded into a time period gap ofthe other moving image. “Mixed time-lapse moving image” is a movingimage in which video pictures of a plurality of channels are switchedalternately along a time-line.

The channel separation processing is executed on a selected frame group.The execution of the channel separation processing needs a plurality offrames. Accordingly, it is presumed that, in a state in which theselected frame group includes only one frame, an operation button or thelike for starting the channel separation processing is disenabled andthe channel separation processing cannot be started. Alternatively, ifthe selected frame group includes only one frame but there are aplurality of frames on an active time-line, all of the frames on theactive time-line may be set to be subjected to the channel separationprocessing. Hereinafter, the frame group to be subjected to the channelseparation processing is referred to as “target frame group”.Furthermore, the moving image that is constituted by a target framegroup is referred to as “target moving image”.

The channel separation processing includes automatic sorting processingfor automatically sorting a plurality of frames included in a targetmoving image by a plurality of channels, and re-sorting processing inwhich a user corrects a sorting result of the automatic sortingprocessing by a manual operation.

The automatic sorting processing is executed by the automatic sortingunit 41. The automatic sorting processing is processing for performingimage processing on a plurality of frames included in a target movingimage so as to automatically sort the frames by a plurality of channels.Specifically, the automatic sorting unit 41 calculates a degree ofsimilarity between frames, and sorts the plurality of frames included inthe target moving image by a plurality of channels based on the degreeof similarity. The degree of similarity between frames is calculated asan index of whether or not the compared frames belong to the samechannel. As a result of the automatic sorting processing, although theautomatic sorting processing will be described later in detail, theplurality of channels are detected. Each channel includes a plurality offrames. Note that as long as a mixed time-lapse moving image issubjected to the automatic sorting processing, a plurality of channelsare usually detected, but there may be a case where only one channel isdetected as a result of the automatic sorting processing. Furthermore,in the automatic sorting processing, a sort label “non-sorted” is givento the frame that is determined as not belonging to any channel.Accordingly, as a result of automatic sorting processing, the channelname of the corresponding channel or the label “non-sorted” is given toeach of the frames included in the target moving image.

The re-sorting processing is executed by the re-sorting unit 45. There-sorting processing is processing for individually sorting anarbitrary frame included in the target moving image to an arbitrarychannel in accordance with a user operation. Specifically, in there-sorting processing, the non-sorted frame that has not been sorted toany channel by the automatic sorting processing can be individuallysorted to a specific channel. Furthermore, the frame that has beensorted to a wrong channel by the automatic sorting processing can alsobe sorted to another correct channel. Accordingly, when the userconsiders the result of the automatic sorting processing as wrong, he orshe can correct the channel to which the problematic frame belongs in aunit of a frame. Furthermore, in the re-sorting processing, it is alsopossible to create a new channel or combine a plurality of channels intoone channel in accordance with a user operation. That is, the re-sortingprocessing is used for manual correction of a result of the automaticsorting processing.

Upon detecting that the user has performed a predetermined operation viathe input unit 20, the automatic sorting unit 41 starts the channelseparation processing. After the start-up of the channel separationprocessing, the setting unit 42 first displays an area setting window W3for displaying a selected frame (see FIG. 5) while overlapping it withthe basic screen W1. The area setting window W3 is a screen foraccepting, from a user, the designation of an area in a selected framesuch that the area serves as a reference for calculating the similaritydegree between frames in the automatic sorting processing. “Selectedframe” refers to one frame included in a target frame group, and may be,for example, the first frame of the target frame group along thetime-line, or the active display frame. Note that although it is clearfrom the description above, the active display frame is the frame thatwas most recently selected on the time-line bar T5, and thus is sure tobe included in the target frame group.

If the target moving image is a video picture from a security camera orthe like, it is often the case that the frame screen has a date and timedisplay area (in the lower left part in the example of FIG. 5). Ifinformation of the entire frame including such a date and time displayarea is used to determine the similarity degree between frames, theremay be a risk that even frames belonging to the same channel aredetermined to have a low degree of similarity. Furthermore, blackframing may be provided in the frame screen. In such a case, conversely,if information regarding the entire frame including the black framing isused to determine the similarity degree of frames, there may be a riskthat even frames belonging to different channels are determined to havea high degree of similarity. Also in a case where the target movingimage is a mixed time-lapse moving image that is constituted by videopictures from security cameras placed in different elevators, the degreeof similarity between frames belonging to different channels is likelyto be determined as being high. This is because the scenes in theelevators are similar to each other. The area setting window W3 avoidssuch a situation, and is useful for excluding the area that may causeerroneous determination from the areas serving as a basis of thesimilarity degree calculation, so that the degree of similarity betweenframes that is to be calculated in the automatic sorting processingserves as a correct index indicating whether or not the frames belong tothe same channel. Specifically, the user designates an area thatcharacterizes the background of each channel in the selected frame ofthe area setting window W3. Taking the elevator as an example, the useronly needs to designate the area in which a poster or the like thatcharacterizes the scene of an elevator is captured. When the userperforms an operation using a mouse or the like to designate anarbitrary area in the selected frame on the area setting window W3, anenclosing line 71 that indicates the area designated by the user isdisplayed in the selected frame. Accordingly, the user can check whetheror not a correct area is designated.

Then, when the user has pressed a start button 72 on the area settingwindow W3, the automatic sorting unit 41 detects this fact and theautomatic sorting processing, which will be described later, starts.Also, at the same time, the area setting window W3 is closed. Note that,when the start button 72 has been pressed in the state in which no areais designated in the selected frame, the entire frame screen is used asa basis of the similarity degree calculation in the automatic sortingprocessing. A cancel button 73 is a button for cancelling areadesignation via the area setting window W3.

During execution of the automatic sorting processing, a dialog box forshowing the progress situation of the automatic sorting processing isdisplayed. Then, when the automatic sorting processing ends, the dialogbox is closed, and the re-sorting unit 45 starts re-sorting processing.The automatic sorting processing will be described later in detail. There-sorting unit 45 first displays a re-sorting window W4 shown in FIG.6, which serves as a user interface for performing re-sorting after theautomatic sorting processing, while overlapping it with the basic screenW1.

A channel list area C1, a frame list area C2, a play area C3, and atime-line bar C4 are arranged on the re-sorting window W4. In thechannel list area C1, channel objects 91 that respectively correspond tochannels detected in the automatic sorting processing are arranged. Inthe example of FIG. 6, four channel objects 91 that correspond to fourchannels CH01, CH02, CH03, and CH04 are sequenced. In the presentembodiment, the channel objects 91 have an icon format.

As shown in FIG. 6, the channel objects 91 respectively indicate thechannel names CH01, CH02, and the like of the corresponding channels.Furthermore, each channel object 91 indicates a thumbnail image of therepresentative frame that represents a plurality of frames belonging tothe corresponding channel. The representative frame is suitably switchedin response to a user operation that will be described later, but thedefault frame is, for example, the first frame, along the time-line, ofthe frames belonging to the corresponding channel. Note that, althoughnot shown in FIG. 6, if the number of channels is so large that thechannel objects 91 corresponding to all of the channels cannot bearranged in the channel list area C1, a scroll bar will appear in thearea C1, and the area C1 will actually be expanded. Furthermore, eachchannel object 91 also indicates the number of frames belonging to thecorresponding channel. The number of frames changes in response to auser operation that will be described later, and the display of thenumber of frames in the channel object 91 will be switched in real timein accordance with the change.

Furthermore, in addition to the channel objects 91, a non-sorted object92 is also arranged in the channel list area C1. The non-sorted object92 is an object that corresponds to the sort in which non-sorted framesthat are included in the target frame group and do not belong anychannel are put together. Note that “the sort in which non-sorted framesare put together” does not correspond to an actual channel, butvirtually constitutes one channel in the sense that the frames are puttogether. Therefore, the non-sorted object 92 has a shape similar tothat of the channel objects 91, and has, in the present embodiment, anicon that has the same size as that of the channel objects 91.Specifically, the non-sorted object 92 displays “non-sorted”, in placeof the channel name, and indicates the number of non-sorted frames, inplace of the number of frames belonging to the corresponding channel.The display of the non-sorted object 92 with the number of frames isalso switched in real time in accordance with the change in the numberof non-sorted frames. Furthermore, also the non-sorted object 92displays a thumbnail image of the representative frame that representsthe “non-sorted” frames. This representative frame is also suitablyswitched in response to a user operation that will be described later,but the default frame is, for example, the first frame, along thetime-line, of the frames belonging to the sort “non-sorted”. In thepresent embodiment, a border line 93 is drawn between the non-sortedobject 92 and the channel objects 91.

The re-sorting unit 45 accepts, from a user, a selection operation forselecting an arbitrary object from among all of the channel objects 91and the non-sorted object 92 in the channel list area Cl. The objectcurrently selected from among all of the objects 91 and 92 is displayedin a different mode from the other objects. In the example of FIG. 6,the object 91 of the channel CH01 is displayed in a color different fromthe other objects 91 and 92. Hereinafter, the channel or sort thatcorresponds to the currently selected one of the objects 91 and 92 isreferred to as “selected channel”.

In the frame list area C2, thumbnail images 94 of all of the framesbelonging to the selected channel are displayed in a list. Note that ifthe number of frames belonging to the selected channel is so large thatall of the thumbnail images 94 cannot be arranged in the frame list areaC2, a scroll bar 95 will appear in the area C2, and the area C2 willactually be expanded. Accordingly, by switching the selection state ofthe objects 91 and 92, the user can selectively display, in the framelist area C2, all of the non-sorted frames or all of the framesbelonging to the corresponding channel in a list. Accordingly, it ispossible to check all of the non-sorted and sorted frames included inthe target frame group while advantageously using the limited space inthe frame list area C2. Particularly, even if a wrong frame is sorted toa channel, the wrong frame will easily be found according to the listdisplay mode. Note that, in the present embodiment, the thumbnail images94 in the channel list area C1 can be sequenced in multiple stages, andare arranged in a time-series order from left to right and from upper tolower with an increase in the number n of the original frame on thetime-line.

The re-sorting unit 45 manages one specific frame belonging to eachchannel as an active frame, and also manages one specific non-sortedframe as an active frame. In the frame list area C2, an enclosing box 96is added to the thumbnail image 94 of the active frame as a mark fordistinguishing this thumbnail image 94 from the thumbnail images 94 ofthe other frames. Each thumbnail image 94 in the frame list area C2 isrecognized as an object, and the user can select, with a click operationor the like, an arbitrary thumbnail image 94 from among all of thethumbnail images 94 in the frame list area C2. Furthermore, the user canalso select a plurality of thumbnail images 94 in the frame list area C2at the same time, by repeatedly performing click operations whilepressing down a specific key on a keyboard, for example. Each time athumbnail image 94 is selected in the frame list area C2, the re-sortingunit 45 switches the active frame of the selected channel to the framethat corresponds to the most recently selected thumbnail image 94. Atthis time, the position of the enclosing box 96 also moves. Furthermore,the active frame is associated with the thumbnail images that areindicated by the channel objects 91 and the non-sorted object 92, andeach time the active frame of each channel or sort is changed, thethumbnail image of the object 91 or 92 that corresponds to the channelor sort is also switched. That is, in the present embodiment, therepresentative frame of each channel or sort corresponds to the activeframe.

The thumbnail image 94 of the active frame in the frame list area C2 canbe individually moved so as to overlap with any of the objects 91 and 92in the channel list area C1 by a drag-and-drop operation. Furthermore,if a plurality of thumbnail images 94 are selected at the same time inthe frame list area C2, the plurality of thumbnail images 94 can bemoved altogether so as to overlap with any of the objects 91 and 92 inthe channel list area C1. (Note that the thumbnail images 94 can be“moved”, but if the thumbnail images 94 are released on the targetobject 91 or 92, they will disappear as if they have been inserted intoa folder.) In other words, the re-sorting unit 45 accepts, from a user,an operation for associating an arbitrary thumbnail image 94 in theframe list area C2 with an arbitrary channel or sort. After theassociating operation, the frame that corresponds to the thumbnail image94 that was moved by this operation is sorted again (re-sorted) to thechannel or sort that corresponds to the object 91 or 92 to which thethumbnail image 94 was moved. Note that if the frame corresponding tothe thumbnail image 94 as per the operation was originally sorted to thechannel or sort that corresponds to the object 91 or 92 as per theoperation, the operation will be ignored and the re-sorting will not beperformed.

As shown in FIG. 6, in the channel list area C1, a new creation object97 is arranged after the sequence of the objects 91 and 92. The newcreation object 97 is an object for creating a new channel, and has anicon format in the present embodiment. Also, the same operation as theabove-described association operation can be performed on not only theobjects 91 and 92 but also the new creation object 97. Specifically, there-sorting unit 45 accepts, from a user, an operation for independentlymoving the thumbnail image 94 of the active frame in the frame list areaC2 so that it overlaps with the new creation object 97 with adrag-and-drop operation. Furthermore, if a plurality of thumbnail images94 in the frame list area C2 are selected at the same time, theplurality of thumbnail images 94 can be moved altogether so as tooverlap with the new creation object 97. (Note that the thumbnail images94 can be “moved”, but if the thumbnail images 94 are released on thenew creation object 97, they will disappear as if they have beeninserted into a folder.) After the association operation, a channelobject 91 is newly created at the position at which the new creationobject 97 was present.

The newly created channel object 91 is an object indicating a newchannel that includes, as an element or elements, a frame or framescorresponding to one or more thumbnail images 94 moved by theassociation operation. In other words, the frame or the frames thatcorresponds or correspond to one or more thumbnail images 94 moved bythe association operation is or are re-sorted to the new channel.Accordingly, if only one thumbnail image 94 was moved, the thumbnailimage 94 will be displayed on the object 91 immediately after thechannel object 91 is newly created. On the other hand, if a plurality ofthumbnail images 94 were moved, one of these thumbnail images 94 (forexample, the most recently selected thumbnail image 94) will bedisplayed on the newly created channel object 91. Also a channel name issuitably given and displayed. Furthermore, the new creation of thechannel object 91 involves movement of the new creation object 97 behindthe sequence of the objects 91 and 92 including the newly created object91.

After the re-sorting of the frame with the above-described associationoperation, the channel name CH01, CH02, or the like or the label“non-sorted” is again given to the re-sorted frame, based on the channelor sort to which the frame is re-sorted. Furthermore, the thumbnailimage 94 of the re-sorted frame is deleted from the frame list area C2.Here, the remaining thumbnail images 94 are aligned in the frame listarea C2 so as to fill up the space in which the deleted thumbnail image94 was arranged. Note that the re-sorted frame or frames does or do notfail to include an active frame. Accordingly, after re-sorting, theactive frame of the selected channel is changed to the nearest one ofthe subsequent frames along the time-line, or to the nearest one of theprior frames along the time-line. The change of the active frame of theselected channel is suitably reflected by the location of the enclosingbox 96, and the indication of the thumbnail image on the object 91 or92. Furthermore, the number of frames belonging to the channel and/orsort that corresponds to the object 91 or 92 from which the re-sortedframe has been moved and to which the re-sorted frame has been moved areagain calculated, and are reflected in the display of the object 91 or92.

Furthermore, the various above-described association operations can berealized by another method than the drag-and-drop operation.Specifically, as shown in FIG. 6, character strings that indicatekeyboard-specific keys are indicated in the lower right parts of theobjects 91, 92, and 97 in the frame list area C2. Upon detecting thatthe specific key indicated in the object 91, 92, or 97 has been pressed,the re-sorting unit 45 determines that the active frame of the currentlyselected channel, that is, the frame to which the enclosing box 96 iscurrently applied in the frame list area C2 has been associated with theobject 91, 92, or 97 that corresponds to the specific key. Theprocessing thereafter is the same as in the case in which theassociation operation is performed using a drag-and-drop operation.

Furthermore, the re-sorting unit 45 can also combine a plurality ofchannels (including the sort “non-sorted” in the present and the nextparagraphs) with each other in response to a user operation.Specifically, in the channel list area C1, an arbitrary object 91 or 92can be individually moved so as to overlap with another object 91 or 92by a drag-and-drop operation. Furthermore, it is also possible that aplurality of objects 91 and 92 are selected at the same time, and aremoved altogether over another object 91 or 92. (Note that the objects 91and 92 can be “moved”, but if the objects 91 and 92 have been releasedon a destination object 91 or 92, they will disappear as if they areinserted into a folder.) In other words, the re-sorting unit 45 accepts,from a user, an operation for associating an arbitrary channel in thelist area C1 with another an arbitrary channel. After the associationoperation, the channel that corresponds to the object 91 or 92 that wasmoved by the operation is combined with the channel that corresponds tothe other object 91 or 92 to which the object 91 or 92 was moved.

After the combination of the channels with the above-describedassociation operation, the label of the channel name, such as“non-sorted”, or CH01, CH02, . . . , of the channel that corresponds tothe destination object 91 or 92 is again given to all of the framesbelonging to the channel that corresponds to the object 91 or 92 thatwas moved. As a result, the destination object 91 or 92 serves as anobject indicating the combined new channel. The active frame of thedestination object 91 or 92 is continuously used as the active frame ofthe combined channel. Furthermore, the re-sorting unit 45 calculates thenumber of frames belonging to the combined channel, and reflects thenumber on the indication of the object 91 or 92. Furthermore, in thechannel list area C1, if a channel object 91 is moved, the moved channelobject 91 disappears, and the remaining objects 91 and 97 are aligned soas to fill up the space in which the channel object 91, havingdisappeared, was arranged. If the non-sorted object 92 is moved, thenumber of frames is zero, but the non-sorted object 92 remains as is.

Furthermore, if there are a plurality of channel objects 91, the orderof the sequence of these objects 91 in the channel list area C1 can bechanged suitably by a drag-and-drop operation. For example, if, in thestate of FIG. 6, the channel object 91 of the channel CH01 is attempt tobe moved to the right of the channel object 91 of the channel CH02, thechannel object 91 of the channel CH01 only needs to be moved between thechannel objects 91 of the channels CH02 and CH03.

Hereinafter, the play area C3 will be described. In the play area C3,the frame group belonging to a selected channel can be reproduced as amoving image. As shown in FIG. 6, a play button 76, a next frame button77, and a previous frame button 78 are arranged on the re-sorting windowW4.

The play button 76 accepts, from a user, an instruction to reproduce theframe group belonging to a selected channel as a moving image. Upondetecting that the play button 76 has been pressed, the re-sorting unit45 displays, in the play area C3, the frames included in the frame groupbelonging to the selected channel sequentially along the time-line in aframe-by-frame format. The reproduction starts from the active frame ofthe selected channel at a point in time when the play button 76 ispressed. Furthermore, the reproduction is performed at the frame rateaccording to the selected channel. Also, during the reproduction, theframe displayed in the play area C3 will be switched sequentially, butat each switch, the active frame managed by the re-sorting unit 45 isupdated to the switched frame in real time. Furthermore, the play button76 accepts, from a user, an instruction to stop the reproduction. Upondetecting that the play button 76 has been pressed during thereproduction, the re-sorting unit 45 fixes the display in the play areaC3 to the active frame at that point in time. Note that also whenstopped, the active frame of the selected channel is constantlydisplayed in the play area C3. Accordingly, when the active frame ischanged during the stop by the operation of selecting a thumbnail image94 in the frame list area C2, the display in the play area C3 is alsochanged in real time.

Furthermore, during the reproduction, the active frame is updated asneeded, and the enclosing box 96 in the frame list area C2 also moves tothe position to enclose the thumbnail image 94 of the most recent activeframe in real time. Furthermore, at this time, the thumbnail image ofthe object 91 or 92 that corresponds to the selected channel is updatedin real time. That is, the display in the play area C3 is synchronizedwith the position of the enclosing box 96 and the display of thethumbnail image of the object 91 or 92. However, another embodiment mayhave a configuration in which no such synchronization is made duringreproduction. In this case, it is also possible to configure such that,for example, first when reproduction is stopped, the position of theenclosing box 96 is moved, and/or, the thumbnail image of the object 91or 92 is updated.

The next frame button 77 and the previous frame button 78 respectivelyaccept, from a user, instructions to switch display in the play area C3to the next frame and to the previous frame, along the time-line, of theframe group belonging to the selected channel. The change of the displayin the play area C3 with the operation using the buttons 77 and 78 isalso associated with the change of the active frame.

The above-described reproduction function is useful for finding a framewrongly sorted to an existing channel. That is, if a frame belonging toa different channel suddenly appears during reproduction of a movingimage, a person can immediately find the frame. In such a case, theperson only needs to immediately stop the reproduction, search for thethumbnail image 94 of the corresponding frame in the frame list area C2,and move the thumbnail image 94 to the correct channel with theabove-described association operation.

A frame rate area C5 for indicating the frame rate of a selected channelis displayed below the play area C3 on the re-sorting window W4. Theframe rate can be calculated by various methods, and can be calculatedas a value obtained by, for example, dividing the number of framesbelonging to the selected channel by a difference between the time ofthe first frame belonging to the selected channel and the time of thefinal frame. Alternatively, the frame rate of the selected channel canalso be calculated based on the following formula:

(The frame rate of the moving image corresponding to the activetime-line)×(the number of frames belonging to the selected channel)÷(thenumber of frames belonging to the active time-line)

However, as described above, the frames belonging to the selectedchannel changes with re-sorting of the frame. Accordingly, each timere-sorting is performed, the re-sorting unit 45 again calculates theframe rate of the selected channel, and changes the display in the framerate area C5 in real time. Furthermore, a channel name area C6 forindicating the channel name or sort name of a selected channel isarranged in the vicinity of the frame rate area C5. Accordingly, a usercan correctly understand to which channel the video picture displayed inthe play area C3 belongs, and to which channel the frame rate displayedin the frame rate area C5 belongs.

Hereinafter, a time-line bar C4 will be described. Similar to thetime-line bar T5 on the basic screen W1, the time-line bar C4 is anobject for schematically showing the active time-line. The target framegroup to be subjected to channel separation processing constitutes apart or whole of the active frame group. It is thus important that theactive time-line can be managed also on the re-sorting window W4, andthus the time-line bar C4 similar to the time-line bar T5 on the basicscreen WI is arranged. The time-line bar C4 extends in the horizontaldirection, and is divided into the number of frames of the active framegroup in the horizontal direction. The n^(-th) divided region from theleft of the time-line bar C4 corresponds to the n^(-th) frame of theactive time-line (n is a natural number).

The re-sorting unit 45 displays, as a mark showing the divided regionthat corresponds to the active frame of the selected channel on thetime-line bar C4, a straight line 83 at the position of the dividedregion. That is, the straight line 83 shows the position, on the activetime-line, of the frame that is displayed in the area C3 and in theenclosing box 96 of the area C2. Furthermore, a target range bar 85 isdisplayed below the time-line bar C4. Similar to the time-line bar C4,the target range bar 85 extends in the horizontal direction over a rangethat corresponds to the segment of the target frame group.

The time-line bar C4 is extendable in the horizontal direction.Specifically, a scale changing bar 86 is arranged on the re-sortingwindow W4, and the scale changing bar 86 is an object into which twoobjects, namely, a slide groove 61 and a slider 62 are combined. Theslide groove 61 has a GUI like a groove extending straightly in thehorizontal direction, and the slider 62 has a GUI that slides in theslide groove 61 in the horizontal direction. By operating the slider 62via the input unit 20, the user can reciprocate the slider 62 along theslide groove 61 in the horizontal direction. The re-sorting unit 45changes the horizontal scale of the time-line bar C4 in a stepwisemanner, based on the position of the slider 62 in the slide groove 61 inthe horizontal direction. Specifically, the horizontal scale of thetime-line bar C4 decreases gradually with the movement of the slider 62to the right side, and the horizontal scale of the time-line bar C4increases in a stepwise manner with the movement of the slider 62 to theleft side. Note that, when the horizontal scale of the time-line bar C4is changed, the sizes in the horizontal direction of all divided regionsare uniformly changed in proportion thereto. Note that, when the slider62 reaches the leftmost position in the slide groove 61, the time-linebar C4 changes to a thumbnail list C7 (see FIG. 7). Furthermore, if thehorizontal scales of the time-line bar C4 and the thumbnail list C7 aretoo long to be accommodated in the re-sorting window W4, a scroll bar 87that scrolls in the horizontal direction will appear, and the area fordisplaying the time-line bar C4 is actually extended in the horizontaldirection.

The thumbnail list C7 is a list in which thumbnail images 88 of all ofthe frames belonging to the active time-line are sequenced in thehorizontal direction. Accordingly, the user can also check, on there-sorting window W4, a frame that is included in the active time-lineand does not belong to the target frame group. The n-^(-th) thumbnailimage 88 from the left on the thumbnail list C7 is the thumbnail imageof the n-^(th) frame on the active time-line (n is a natural number). Inthe thumbnail list C7, an enclosing box 89 is added to the thumbnailimage 88 of the active frame of the selected channel, the enclosing box89 serving as a mark for distinguishing this thumbnail image 88 from theother thumbnail images 88.

Similar to the time-line bar T5, the time-line bar C4 and the thumbnaillist C7 accept, from a user, the selection of an arbitrary frame on theactive time-line. In other words, the user can select a divided regionon the time-line bar C4 or an arbitrary thumbnail image 88 in thethumbnail list C7 with a click operation or the like, so as to select anarbitrary frame from among the active frame group. Also, each time aframe is selected on the time-line bar C4 and the thumbnail list C7, there-sorting unit 45 determines whether or not the frame is included inthe target frame group. If it is determined that the frame is included,the channel or sort to which the frame belongs is switched to theselected channel. At the same time, the re-sorting unit 45 switches theframe to the active frame of the selected channel. Here, the change inthe selected channel and the active frame is reflected on the display inthe channel list area C1, the frame list area C2, the play area C3, theframe rate area C5, and the channel name area C6 in real time.Furthermore, the positions of the straight line 83 and the enclosing box89 are also changed in real time.

Using the above-described various functions, the user can performre-sorting of a plurality of frames included in the target frame group.At the time when it is determined that the re-sorting has ended, an OKbutton E5 is pressed. In response thereto, the re-sorting unit 45 closesthe re-sorting window W4 and newly creates at least one moving imagebelonging to a new time-line. The at least one moving image that isnewly created has a one-to-one correspondence with the channel that wasfinally defined on the re-sorting window W4 and was selected by the user(hereinafter, referred to as “final channel”). As shown in FIG. 6, theselection of the final channel is made by the user checking a mark in aselection box 98 arranged on the channel object 91. Note that, in thepresent embodiment, the default value of the selection box 98 is“selected”. Accordingly, only with respect to the channel for which itis determined that no moving image creation is needed, the user justneeds to operate the selection box on the corresponding object 91 so asto change the state to “non-selected”. Furthermore, in the presentembodiment, a selection box 99 is also arranged below the play area C3.The selection box 99 is an object for selecting whether or not themoving image of the selected channel (note that the sort “non-sorted” isnot included therein) needs to be created, and is synchronized with theselection box 98 on the channel object 91. The re-sorting unit 45 checksthe selection states of the selection boxes 98 and 99, and decides thefinal channel.

A moving image that is newly created in the above-described manner issuch that all of the frames belonging to the corresponding final channelare sequenced along the new time-line. Also, at this time, the displaywindow W2 that has a one-to-one correspondence with the new moving imageis newly created. Furthermore, the re-sorting unit 45 creates a stillimage file group that has a one-to-one correspondence with the framegroup included in this new moving image, stores the created still imagefile group in the processed file region 52, and handles it as the0-order frame. That is, thereafter, the new moving image can beprocessed similarly to a still image file group loaded in the originalimage region 51. Specifically, the new moving image can similarly bereproduced in the display window W2, and is similarly to be subjected tovarious image processing.

Note that the moving image that corresponds to the sort “non-sorted” isnot newly created. Accordingly, it is also possible that, at the timewhen the OK button E5 has been pressed, the re-sorting unit 45determines whether or not there remains a non-sorted frame, and if it isdetermined that there remains a non-sorted frame, the user is notifiedof this fact. In this case, if it is determined that there remains anon-sorted frame, the re-sorting unit 45 displays a check window W5 asshown in FIG. 8 on the display 10. When a “Yes” button 74 is pressed,the procedure moves to the above-described processing for newly creatinga moving image. When a “No” button 75 is pressed, the check window W5 isclosed, and the display returns to the re-sorting window W4. On theother hand, if it is determined that there remains no non-sorted frame,the procedure immediately moves to the above-described processing fornewly creating a moving image.

<1-4. Automatic Sorting Algorithm>

Hereinafter, the automatic sorting processing will be described indetail with reference to FIG. 9. As described above, the automaticsorting processing starts when the start button 72 on the area settingwindow W3 has been pressed. In the following, a plurality of framesconstituting a target frame group are denoted by F₁, F₂, . . . , F_(J)(where J is an integer of 2 or more). Note that the sequence of theframes F₁, F₂, . . . , F_(J) is equivalent to the sequence on thetime-line.

First, in step S1, the automatic sorting unit 41 scales down the size ofthe frames F₁, F₂, . . . , F_(J) included in a target frame group. Atthis time, the size of the frames may be scaled down so that the numberof the lateral and/or vertical pixels of the frames is a predeterminednumber, or so that the frames have a size that accounts for apredetermined ratio of the original size. Furthermore, at this time, anaspect ratio may or may not be saved. With this scale down step, thefollowing processing is accelerated, and the influence of noise isreduced. Hereinafter, the scaled-down frames F₁, F₂, . . . , F_(J) arealso denoted by F₁, F₂, . . . , F_(J).

In the following step S2, the setting unit 42 defines a channel CH₁, andsorts the frame F₁ to the channel CH₁. Specifically, the label channelname “CH₁” is given to the frame F₁. Furthermore, the setting unit 42sets the frame F₁ as a reference frame G₁, which represents the framegroup that belongs to the channel CH₁. Note that by step S10, which willbe described later, the number of channels can be increased as needed.Also, for a channel CH₁ that is to be newly created, a reference frameG₁ (1=2, 3, . . . ), which represents the frame group that belongs tothe corresponding channel, will be set.

After the above-described processing has ended, the setting unit 42sequentially selects, from among the remaining frames F₂, F₃, . . . ,F_(J), a frame F_(j) (j=2, 3, . . . , J) along the time-line, and setsthe frame F_(j) as a comparative frame. Meanwhile, each time acomparative frame F_(j) is set, the calculation unit 43, thedetermination unit 44, and the setting unit 42 repeatedly performfollowing steps S3 to S10 on the comparative frame F_(F), and performcomparison with the existing reference frames G₁, G₂, . . . , G_(L) (Lis the number of existing channels).

In step S3, the calculation unit 43 divides the entire comparative frameF_(j) into a predetermined number K (K is an integer of 2 or more) ofsub-regions (blocks) D₁, D₂, . . . , D_(K) (see FIG. 10). Note that“entire comparative frame F_(j)” in this context refers to, if an areais designated on the area setting window W3, the entire area, whereas,if no designation was made, the entire frame screen. Note that thesub-regions D₁, D₂, . . . , D_(K) are not necessarily regions obtainedby thoroughly dividing the entire comparative frame F_(j), and maypartially overlap with each other. The shape thereof is not necessarilya rectangle, and may be, for example, a circle or another polygon.Furthermore, the sub-regions may have a uniform shape and size, ordifferent shapes and sizes.

In subsequent step S4, the calculation unit 43 divides each of thereference frames G₁, G₂, . . . , G_(L) of the existing channels CH₁,CH₂, . . . , CH_(L) into sub-regions D′₁, D′₂, . . . , D′_(K) (see FIG.10). The sub-regions D′₁, D′₂, . . . , D′_(K) are respectively definedso as to be located at the same positions as those of the sub-regionsD₁, D₂, . . . , D_(K) in the frame screen.

Then, in step S5, the calculation unit 43 calculates, for eachcombination of the sub-regions D_(k) and D′_(k) (k=1, 2, . . . , K), asimilarity degree Y_(lk) ((hereinafter, referred to as “local similaritydegree”) between the comparative frame F_(j) and each reference frameG_(l) (l=1, 2, . . . , L). That is, the local similarity degree Y_(lk)(refers to the similarity degree between the sub-region D_(k) of thecomparative frame F_(F) and the sub-region D′_(k) of the reference frameG_(l). In the present embodiment, a correlation coefficient that iscalled “zero-mean normalized cross correlation” (ZNCC) is calculated,but in another embodiment, another similarity degree such as acorrelation coefficient that is called “NCC”, or a similarity degreethat is called “SDD” or “SAD” may be calculated.

Then, in step S6, the calculation unit 43 calculates a frame-overallsimilarity degree B₁ (hereinafter, referred to as “total similaritydegree”) between the comparative frame F_(F) and the correspondingreference frame G_(l) (l=1, 2, . . . , L). Specifically, the calculationunit 43 determines, for each l (l=1, 2, . . . , L), the sub-regionsD_(k) and D′_(k) that have a high local similarity degree Y_(lk.) (Atthis time, for example, the local similarity degree Y_(lk)(may bedetermined as high if it is within a predetermined high order range, ormay be determined as high if it is larger than a predeterminedthreshold. Then, the calculation unit 43 again calculates, for each l(l=1, 2, . . . , L), the similarity degree between the comparative frameF_(j) and the reference frame G_(l) only using the sub-regions D_(k) andD′_(k) that have a high local similarity degree Y_(lk) (in the entireframe, and sets the calculated similarity degree as the total similaritydegree B_(l) (see FIG. 10). In the present embodiment, a zero-meannormalized cross correlation (ZNCC) is also calculated as the totalsimilarity degree B_(l), but another similarity degree may becalculated, similar to the case of the local similarity degree Y_(lk).Furthermore, the total similarity degree B_(l) and the local similaritydegree Y_(lk) may be calculated using different methods.

Accordingly, in the present embodiment, the information relating to thesub-regions D_(k) and D′_(k) that have a low local similarity degreeY_(lk) (is not taken into consideration for the calculation of the totalsimilarity degree B_(l), and the reason thereof is as follows. That is,frames belonging to the same channel have the same background imageexcept for a moving subject part. If the moving subject part accountsfor a large ratio in the entire frame, and the total similarity degreeB_(l) is determined using the information relating to the entire frame,even frames belonging to the same channel will have a low totalsimilarity degree B_(l). Accordingly, here, the sub-regions D_(k) andD′_(k) that are considered to include a large moving subject part andhave a low local similarity degree Y_(lk) (are not used in thecalculation of the total similarity degree B_(l), in order to preventerroneous determination. As a result, the influence of a moving subjectis reduced, and it is possible to correctly determine whether or notcompared frames belong to the same channel.

In subsequent step S7, the determination unit 44 determines the maximumof the total similarity degrees B₁, B₂, . . . , B_(L) calculated in stepS6, and determines whether or not the maximum total similarity degree islarger than a predetermined threshold. If it is determined that themaximum total similarity degree is larger than the predeterminedthreshold, the procedure advances to step S8, whereas if it isdetermined that the maximum total similarity degree is the predeterminedthreshold or less, the procedure advances to step S10.

In step S8, the determination unit 44 determines that the comparativeframe F_(j) belongs to a channel CH_(MAX). “MAX” is a value of “1” ofthe maximum total similarity degree B_(l). Specifically, thedetermination unit 44 gives the channel name label “CH_(MAX)” to thecomparative frame F), and sorts the comparative frame F_(j) to thechannel CH_(MAX). Accordingly, the comparative frame F_(j) will belongto the same channel as that of the frame group belonging to the channelCH_(MAX).

After step S8, the procedure advances to step S9. In step S9, thesetting unit 42 updates a reference frame G_(MAX) of the channelCH_(MAX). Specifically, the setting unit 42 combines the comparativeframe F_(j) with the existing reference frame G_(MAX) into a newreference frame G_(MAX). In the present embodiment, weighted averagingis employed as a combination method, in which a larger weight is givento the comparative frame F_(j) than the existing reference frameG_(MAX). Accordingly, the reference frame G_(MAX) that represents theframe group belonging to the channel CH_(MAX) is a combined image inwhich a larger weight is given to a nearer frame. As a result, whendetermining whether or not frames belong to different channels, it ispossible to support a time-series change. Note that in otherembodiments, the comparative frame F_(j) may directly be set as a newreference frame G_(MAX). Alternatively, step S9 may be omitted so thatan update of the reference frame G_(MAX) is not performed.

On the other hand, step S10 is a step that is executed when all of thetotal similarity degrees B₁, B₂, . . . , B_(L) calculated in step S6 aredetermined as being a predetermined threshold or less. In other words,step S10 is a step that is executed when the comparative frame F_(j) isnot similar to the reference frame G_(l) of any of the channels CH₁. Instep S10, the setting unit 42 defines a new channel CH_(L+1), inaddition to the existing channels CH₁, CH₂, . . . , CH_(L), and sortsthe comparative frame F_(j) to the channel CH_(L+1). Specifically, a newchannel name label “CH_(L+1)” is given to the comparative frame F_(j).Furthermore, the setting unit 42 sets the comparative frame F as areference frame G_(L+1), which represents the frame group belonging tothe new channel CH_(L+1).

When steps S3 to S10 on all of the frames F₁, F₂, . . . , F_(j) haveended, the procedure advances to step S11. In step S11, the existingchannels CH₁, CH₂, . . . , CH_(L) are corrected. Specifically, thecalculation unit 43 calculates the frame-overall similarity degreebetween the reference frames G₁, G₂, . . . , G_(L) in an all-to-allmanner. Note that, similar to step S3, “frame-overall” in this contextrefers to, if an area is designated on the area setting window W3, theentire area, whereas, if no designation was made, the entire framescreen. Furthermore, in the present embodiment, a zero-mean normalizedcross correlation (ZNCC) is calculated as the similarity degree butanother similarity degree may be calculated, similar to steps S5 and S6.Furthermore, the similarity degree may be calculated by a methoddifferent from methods by which the total similarity degree B₁ and thelocal similarity degree Y_(lk) are calculated. Also, if such asimilarity degree exceeds a predetermined threshold, the determinationunit 44 combines the channels CH₁ that correspond to the referenceframes G₁ that give such a similarity degree into one channel.Specifically, the same channel name label is again given to all of theframes F_(j) belonging to the channels CH₁ to be combined. Note herethat three or more channels CH_(I) may be combined into one.

Furthermore, if there is a channel CH₁ that includes only one frame evenafter the above-described combination of channels, the determinationunit 44 deletes the channel CH₁. Specifically, the determination unit 44again gives the label “non-sorted” to all of the frames F_(j) belongingto the channel CH₁ to be deleted. Note that, in the present embodiment,the channel CH₁ that includes only one frame F_(j) is to be deleted, butsuch a reference value may be two, three, or the like. Furthermore, itis also possible to set an upper limit for the number of channels thatcan be created. In this case, deletion can be started from the channelthat includes the smallest number of frames F_(j), so that the number ofchannels does not exceed the upper limit.

According to the above-described processing, it is likely that thevalues included in the channel names CH₁ of the finally created channelsare not a consecutive number. Accordingly, the determination unit 44again gives the channel names such as CH₀₁, CH₀₂, and the likesequentially to the finally created channels CH₁, and updates the labelsgiven to the frames F_(j) to these new channel names. Then, theautomatic sorting processing ends.

<2. Second Embodiment>

Hereinafter, a second embodiment of the present invention will bedescribed. FIG. 11 shows a flow of automatic sorting processingaccording to the second embodiment, and the second embodiment differsfrom the first embodiment in an automatic sorting algorithm.Furthermore, as is clear from the comparison between FIG. 9 and FIG. 11,a difference between both of the automatic sorting processes is that,instead of steps S4 to S6, steps S24 to S26 are inserted in the secondembodiment. Accordingly, in the following, for ease of description, onlythe difference will be described.

In the second embodiment, after execution of steps S1 to S3 as with thefirst embodiment, step S24 is executed. In step S24, the calculationunit 43 detects feature points in each of the sub-regions D₁, D₂, . . ., D_(K) of the comparative frame F_(j). At that time, the numbers of thefeature points that are detected in the respective sub-regions D₁, D₂, .. . , D_(K) are preferably the same. Hereinafter, the feature points ofthe comparative frame F_(j) are denoted as P₁, P₂, . . . , P_(U) (U isan integer of 2 or more).

The calculation unit 43 sets, in the comparative frame F_(j), asub-region V_(u) in the vicinity of each feature point P_(u) (u=1, 2, .. . , U) (see FIG. 12). The sub-region V_(u) is a region of apredetermined size in which the feature point P_(u) is located at thecenter. Then, the calculation unit 43 sets sub-regions V′₁, V′₂, . . . ,V_(u) for each of the reference frames G₁, G₂, . . . , G_(L) of theexisting channels CH₁, CH₂, . . . , CH_(L) (see FIG. 12). Thesub-regions V′₁, V′₂, . . . , V′_(U) are respectively defined so as tobe located at the same positions as those of the sub-regions V₁, V₂, . .. , V_(U) in the frame screen.

As described above, in the present embodiment, the feature points P₁,P₂, . . . , P_(U) are detected from the comparative frame Fj on a basisof the sub-regions D₁, D₂, . . . , D_(K). As a result, the featurepoints P₁, P₂, . . . , P_(u) are detected substantially uniformly fromthe entire comparative frame F_(j) without being biased to a partialregion of the entire screen of the comparative frame F_(j).

Then, in step S25, the calculation unit 43 calculates, for eachcombination of sub-regions V_(u) and V′_(u) (u=1, 2, . . . , U), thelocal similarity degree Y_(lu) between the comparative frame F_(j) andeach reference frame G_(l) (l=1, 2, . . . , L). That is, “localsimilarity degree Y_(lu)” refers to the similarity degree between thesub-region V_(u) of the comparative frame F_(j) and the sub-regionV′_(u)of the reference frame G_(l). Note that the local similaritydegrees in the first and second embodiments are common in that theydenote the similarity degrees between sub-regions, and thus the samereference sign Y is used to denote the local similarity degrees.Furthermore, in the present embodiment, a correlation coefficient,called a zero-mean normalized cross correlation (ZNCC), is calculated,but another similarity degree may be calculated, similar to the firstembodiment.

Then, in step S26, the calculation unit 43 calculates the frame-overalltotal similarity degree B_(l) between the comparative frame F_(j) andthe corresponding reference frame G_(l) (l=1, 2, . . . , L). Note thatthe total similarity degrees in the first and second embodiments arecommon in that they denote the similarity degrees in the entire frame,and thus the same reference sign B is used for denoting the totalsimilarity degrees. Specifically, the calculation unit 43 determines,for each l (l=1, 2, . . . , L), the sub-regions V_(u) and V′_(u) thathave a high local similarity degree Y_(lu). At this time, for example,the local similarity degree may be determined as high if the localsimilarity degree Y_(lu) is in a predetermined high order range, or maybe determined as high if the local similarity degree Y_(lu) is largerthan a predetermined threshold. Then, for each l (l=1, 2, . . . , L),the similarity degree between the comparative frame F_(j) and thereference frame G_(l) is again calculated only using the sub-regionsV_(u) and V′_(u) of the entire frame that have a high local similaritydegree Y_(lu), and the calculated similarity degree is set as the totalsimilarity degree B_(i). Also in the present embodiment, a zero-meannormalized cross correlation (ZNCC) is calculated as the totalsimilarity degree B₁, but another similarity degree may be calculatedsimilar to the first embodiment. Also in the present embodiment, thetotal similarity degree B_(l) and the local similarity degree Y_(lu) maybe calculated by different methods.

With step S26 above, the total similarity degrees B₁, B₂, . . . , B_(L)are calculated. The flow of the processing thereafter is the same as inthe first embodiment.

As described above, in the present embodiment, information relating toregions other than the sub-regions V₁, V₂, . . . , V_(U) in the vicinityof the feature points P₁, P₂, . . . , P_(U) is not taken intoconsideration for the calculation of the total similarity degrees. Thisis because, if frames, even of different channels, are of imagesobtained by capturing similar scenes, and information relating to theentire frame is used for determining the total similarity degree B₁, ahigh total similarity degree B₁ will be obtained. For example, in a caseof two video pictures in which different places of the same shopinterior are captured, the major part of the frame of both videopictures may include wallpaper, a floor, and the like in the sameuniform color, and in such a case, the total similarity degree B₁ ishigh. Accordingly, here, the regions in which a background such aswallpaper and a floor in the same uniform color is likely to becaptured, that is, the regions that are not located in the vicinity ofthe feature points P₁, P₂, . . . , P_(U) are not used in the calculationof the total similarity degree B₁, in order to prevent erroneousdetermination. As a result, it is possible to reduce the influence of asimilar background, and to correctly determine whether or not comparedframes belong to the same channel.

Furthermore, in the present embodiment, information relating to thesub-regions V_(u) and V′_(u) that have a low local similarity degreeY_(lu) is also not taken into consideration for the calculation of thetotal similarity degree B₁. This is because, if information relating tothe entire frame is used to determine the total similarity degree B₁when the ratio of a moving subject part in the entire frame is large,frames, even of the same channel, will have a low total similaritydegree B₁. Accordingly, here, the sub-regions V_(u) and V′_(u) that havea low local similarity degree Y_(lu) and that are considered to includea large moving subject part are not used for the calculation of thetotal similarity degree B₁, in order to prevent erroneous determination.As a result, it is possible to reduce the influence of the movingsubject, and to more correctly determine whether or not compared framesbelong to the same channel.

<3. Third Embodiment>

Hereinafter, a third embodiment according to the present invention willbe described. FIG. 13 shows a flow of automatic sorting processingaccording to the third embodiment, and the third embodiment differs fromthe first and second embodiments only in an automatic sorting algorithm.Furthermore, as is clear from the comparison between FIG. 13 and FIG. 9,a difference between automatic sorting processes according to the firstand third embodiments is only that, instead of steps S5 and S6, subroutine S30 is inserted in the third embodiment. Accordingly, in thefollowing, for ease of description, only the difference will bedescribed.

As described above, in the first and second embodiments, a correlationcoefficient is calculated as a similarity degree between frames, but inthe third embodiment, the similarity degree between frames is evaluatedbased on color trends of the frames. Furthermore, as the color trend ofa frame, an overall color trend in the screen of the frame and a localcolor trend in the screen of the frame are comprehensively taken intoconsideration. In the following, a specific description will be given.

First, in the third embodiment, steps S1 to S4 as with the firstembodiment are executed, and then sub routine S30 is executed. Subroutine S30 is processing for evaluating various color indexesindicating local or overall color trends in the frame screen, and isshown in detail in FIG. 17. Particularly, density is considered in stepsS31 to S33 included in sub routine S30, chroma is considered in stepsS34 to S38, and hue is considered in steps S39 to S43. Note thatalthough density is considered in steps S31 to S33, intensity orbrightness may be used instead of density. Furthermore, frame-overallimpression is considered in steps S44 to 46 included in sub routine S30.

First, in step S31, the calculation unit 43 calculates an averagedensity Hd of the entire comparative frame F_(j). Note that “entirecomparative frame F_(j)” in this context refers to, if an area isdesignated on the area setting window W3, the entire area, whereas, ifno designation was made, the entire frame screen. Furthermore, thecalculation unit 43 calculates an average density Ed_(k) of eachsub-region D_(k) (k=1, 2, . . . , K) of the comparative frame F_(j).Note that “average density” is calculated as an average of the densitiesof all pixels in the target area, and the density of each pixel iscalculated as an average of RGB values of the pixel. Then, thecalculation unit 43 gives the value “1” to each sub-region D_(k) (k=1,2, . . . , K) if density Ed_(k)>density Hd is satisfied, and gives thevalue “0” if density Ed_(k)≦density Hd is satisfied (see FIG. 14). Notethat the values “1” and “0” are indexes that indicate whether thedensity of a local region is relatively high or low relative to theentire frame, and are color indexes that indicate the local color trendregarding density (brightness) (hereinafter, referred to as “densityindex”).

In subsequent step S32, the calculation unit 43 executes the sameprocessing as step S31 on each of the reference frames G₁, G₂, . . . ,G_(L) of the existing channels CH₁, CH₂, . . . , CH_(L). That is, thecalculation unit 43 calculates, for each l (l=1, 2, . . . , L), adensity index with respect to each sub-region D′_(k) (k=1, 2, . . . , K)in the reference frame G₁ (see FIG. 14). Note that, as shown in FIG. 13and FIG. 17, step S32 will be repeatedly executed, and thus, for thereference frame G₁ for which the density index has been calculated, thisvalue is referenced and new calculation processing is omitted.

In subsequent step S33, the calculation unit 43 calculates, based on thedensity indexes calculated in steps S31 and S32, a similarity degree Bd₁between the comparative frame F_(j) and each reference frame G_(l) (l=1,2, . . . , L). Specifically, the calculation unit 43 calculates, as thesimilarity degree Bd_(l), a similarity degree between a density indexdistribution pattern in the comparative frame F_(j) and a density indexdistribution pattern of each reference frame G_(l). In the presentembodiment, the number of combinations of the sub-regions D_(k) andD′_(k) whose density indexes “1” and “0” do not match is counted, andthe square value of the value obtained by dividing this count value by asub-region count K is set as the similarity degree Bd_(l) (see FIG. 14).Note here that the reason why division using the sub-region count K isperformed is to maintain the similarity degree Bd_(l) as a value between0 and 1, and to normalize it. Furthermore, the smaller the similaritydegrees calculated in steps S33, and steps S38 and S43, which will bedescribed later, the higher the similarity is. In this sense, thesesimilarity degrees can be handled as “non-similarity degrees”.

Meanwhile, a composition of an image is decided by arrangement of aplurality of elements in the screen. Many images, if simplified, have acomposition in which a region showing an attention object and a regionshowing the background thereof are broadly separated. That is, theregion having the density index “1” and the region having the densityindex “0” can be considered such that one shows an attention object andthe other shows the background thereof. Based on this consideration, itcan be said that the density index distribution pattern in thecomparative frame F_(j) refers to the composition of the comparativeframe F_(j), and the density index distribution pattern in the referenceframe G₁ refers to the composition of the reference frame G_(l).Accordingly, it can be said that the similarity degree Bd_(l) refers tothe similarity degree of the composition between the comparative frameF_(j) and the reference frame G_(l).

In subsequent step S34, the calculation unit 43 divides, based on thedensity index distribution pattern in the comparative frame F_(j), thecomparative frame F_(j) into two sub-regions R₁ and R₂ (see FIG. 15).The sub-regions R₁ are regions obtained by combining all of thesub-regions D_(k) whose density index is “1” in the comparative frameF_(j), and the sub-regions R₂ are regions obtained by combining all ofthe sub-regions D_(k) whose density index is “0” in the comparativeframe F_(j). That is, in step S34, the comparative frame F_(j) isdivided into the region showing the attention object and the regionshowing the background thereof, based on the composition of thecomparative frame F_(j).

In subsequent step S35, similar to step S34, the calculation unit 43divides, based on the density index distribution pattern of each of thereference frame G₁, G₂, . . . , G_(L) of the existing channels CH₁, CH₂,. . . , CH_(L), each of the reference frame G₁, G₂, . . . , G_(L) intosub-regions R′₁ and R′₂ (see FIG. 15). The sub-region R′₁ is a regionobtained by combining all of the sub-regions D′_(k) whose density indexis “1” in the reference frame G₁, and the sub-region R′₂ is a regionobtained by combining all of the sub-regions D′_(k) whose density indexis “0” in the reference frame G₁. That is, in step S35, the referenceframe G_(l) is divided into the region showing an attention object andthe region showing the background thereof, based on the composition ofthe reference frame G₁.

In subsequent step S36, the calculation unit 43 calculates, for each ofthe sub-regions R₁ and R₂ in the comparative frame F_(j), three averagesfor each of RGB. Then, the calculation unit 43 calculates a chroma ofthe sub-region R₁ based on the three averages for each of RGB in thesub-region R₁, and calculates a chroma of the sub-region R₂ based on thethree averages for each of RGB in the sub-region R₂. Then, thecalculation unit 43 calculates a chroma that is relative between thesub-regions R₁ and R₂ (hereinafter, referred to as relative chroma). Therelative chroma is calculated as the absolute value of a difference inchroma between the sub-region R₁ and the sub-region R₂. The relativechroma is a color index indicating the color trend regarding chroma(hereinafter, referred to as chroma index).

In subsequent step S37, the calculation unit 43 executes the sameprocessing as step S36 on each of the reference frames G₁, G₂, . . . ,G_(L) of the existing channels CH₁, CH₂, . . . , CH_(L). That is, thecalculation unit 43 calculates, for each l (l=1, 2, . . . , L), a chromaof each of the sub-regions R′₁ and R′₂ in the reference frame G₁, andcalculates the relative chroma, which is the absolute value of adifference between these. Note that, as shown in FIG. 13 and FIG. 17,step S37 will be repeatedly executed, and thus, for the reference frameG₁ for which the relative chroma has been calculated, this value isreferenced and new calculation processing is omitted. Step S35 may alsobe omitted similarly.

In subsequent step S38, the calculation unit 43 calculates a similaritydegree Bs_(l) between the relative chroma of the comparative frame F andthe relative chroma of each reference frame G₁ (l=1, 2, . . . , L),which were calculated in steps S36 and S37. In the present embodiment,the similarity degree Bs_(l) is calculated as the square value of thevalue obtained by dividing a difference between both relative chromas by255. Note here that the reason why division using 255 is performed is tomaintain the similarity degree Bs_(l) between 0 and 1, and to normalizeit.

In subsequent step S39, the calculation unit 43 divides the comparativeframe F_(j) into two regions, namely, a main region O₁ and a sub regionO₂. Specifically, of the sub-regions R₁ and R₂ calculated in step S34,the region having a larger area is set as the main region O₁, and theregion having a smaller area is set as the sub region O₂ (see FIG. 16).That is, in step S39, the comparative frame F_(j) is divided into theregion showing an attention object and the region showing the backgroundthereof, based on the composition of the comparative frame F_(j).

In subsequent step S40, similar to step S39, the calculation unit 43divides each of the reference frames G₁, G₂, . . . , G_(L) into tworegions, namely, a main region O′₁ and a sub region O′₂. Specifically,of the sub-regions R′₁ and R′₂ calculated in step S35, the region havinga larger area is set as the main region O′₁, and the region having asmaller area is set as the sub region O′₂ (see FIG. 16). That is, instep S40, the reference frame G₁ is divided into the region showing anattention object and the region showing the background thereof, based onthe composition of the reference frame G₁.

In subsequent step S41, the calculation unit 43 calculates an averagehue of the main region O₁, and calculates an average hue of the subregion O₂. Note that these average hues are each color index indicatinga local color trend regarding hue (hereinafter, referred to as “hueindex”), and are each calculated as an average of hues of all pixels inthe target area.

In subsequent step S42, the calculation unit 43 executes the sameprocessing as step S41 on each of the reference frames G₁, G₂, . . . ,G_(L) of the existing channels CH₁, CH₂, . . . , CH_(L). That is, thecalculation unit 43 calculates, for each l (l=1, 2, . . . , L), a hueindex of each of the regions O′₁ and O′₂ in the reference frame G₁. Notethat, as shown in FIG. 13 and FIG. 17, step S42 will be repeatedlyexecuted, and thus, for the reference frame G₁ for which the hue indexhas been calculated, this value is referenced and new calculationprocessing is omitted. Step S40 may also be omitted similarly.

In subsequent step S43, the calculation unit 43 calculates a similaritydegree Bh_(1l) between the hue index of the main region O₁ in thecomparative frame F_(j), and the hue index of the main region O′₁ ineach reference frame G₁ (l=1, 2, . . . , L). In the present embodiment,the similarity degree Bh_(1l) is calculated as the square value of thevalue obtained by dividing the difference between the hue indexes of themain regions O₁ and O′₁ by 180. Furthermore, the calculation unit 43calculates a similarity degree Bh_(2l) between the hue index of the subregion O₂ in the comparative frame F_(j), and the hue index of the subregion O′₂ in each reference frame G₁ (l=1, 2, . . . , L). Specifically,the square value of the value obtained by dividing the differencebetween the hue indexes of the sub regions O₂ and O′₂ by 180 is set asthe similarity degree Bh_(2l). Note here that the reason why divisionusing 180 is performed is to maintain the similarity degrees Bh₁₁ andBh₂₁ between 0 and 1, and to normalize them.

Subsequent steps S44 to S46 are steps for evaluating a color indexindicating an overall color trend in the frame screen. In the presentembodiment, various impressions Z₁, Z₂, . . . , Z_(I) (I is an integerof 2 or more) are defined in order to evaluate the overall color trendin the frame screen. Note that various types of information for definingthese impressions Z₁, Z₂, . . . , Z_(I) are stored in the impressiondefining region 53 of the software management region 50.

Hereinafter, the impression Z_(i) (i=1, 2, . . . , I) will be described.Each impression Z₁ is associated with one or more color conditions, andweights are defined for the color conditions. Note that the weights ofone or more color conditions that are associated with the sameimpression Z_(i) is summed up to 1. FIG. 18 shows an example of theimpression Z₁ “natural”, and the impression Z₁ “natural” is associatedwith three color conditions “green”, “brown”, and “beige”. Furthermore,the weights 0.5, 0.25, 0.25 are respectively given to the three colorconditions “green”, “brown”, and “beige”. As shown in FIG. 18, for eachof the color conditions, evaluation values for the values of density(intensity), chroma, and hue are defined. Here, in a case of a pixel ofinterest, by obtaining evaluation values for the values of density,chroma, and hue of the pixel and then multiplying these evaluationvalues by each other, the value of each color condition is calculated.Also, the value of the impression Z₁ of the pixel is calculated as avalue obtained by giving the above-described weights to the values ofthe color conditions and multiplying these by each other.

In step S44, the calculation unit 43 calculates the value of eachframe-overall impression Z_(i) (i=1, 2, . . . , I) of the comparativeframe F_(j). Note that “frame-overall” in this context refers to, if anarea is designated on the area setting window W3, the entire area,whereas, if no designation was made, the entire frame screen.Specifically, the calculation unit 43 calculates, for each i (i=1, 2, .. . , I), the value of the impression Z_(i) for each pixel included inthe entire comparative frame F_(j), calculates an average of thesevalues, and sets the average as the value of the frame-overallimpression Z₁. Note that the value of the frame-overall impression Z_(i)is a color index indicating a frame-overall color trend regarding theimpression Z_(i) (hereinafter, referred to as “impression index”).

In subsequent step S45, the calculation unit 43 executes the sameprocessing as step S44 on each of the reference frames G₁, G₂, . . . ,G_(L) of the existing channel CH₁, CH₂, . . . , CH_(L). That is, thecalculation unit 43 calculates, for each l (l=1, 2, . . . , L), animpression index of the reference frame G₁ with respect to each of theimpressions Z₁, Z₂, . . . , Z_(I). Note that, as shown in FIG. 13 andFIG. 17, step S45 is repeatedly executed, and thus, for the referenceframe G₁ for which the impression index has been calculated, this valueis referenced and new calculation processing is omitted.

In subsequent step S46, the calculation unit 43 calculates aframe-overall similarity degree Bi_(l) between the comparative frameF_(j) and each reference frame G₁ (l=1, 2, . . . , L), based on theimpressions Z₁, Z₂, . . . , Z_(I). Specifically, the calculation unit 43calculates, for each l (l=1, 2, . . . , L) and each i (i=1, 2, . . . ,I), the square value of the difference between the value of theframe-overall impression Z_(i) of the comparative frame F_(j) and theframe-overall impression Z_(i) of the reference frame G₁. Then, for eachl (l=1, 2, . . . , L), the calculation unit 43 obtains a value bysubtracting the square root of the sum of I square values (the distancebetween the comparative frame F_(j) and the reference frame G₁ in theI-order impression space) from 1, and sets the resulting value as thesimilarity degree Bi_(l).

Then, in step S47, the calculation unit 43 calculates, based on thealready calculated similarity degrees Bd_(l), Bs_(l), Bh_(1l), Bh_(2l),and Bi_(l) (l=1, 2, . . . , L), a similarity degree B₁ between thecomparative frame F_(j) and each reference frame G₁ (l=1, 2, . . . , L)(hereinafter, referred to as “total similarity degree”). Note that thetotal similarity degrees in the first and third embodiments are commonin that they denote the similarity degrees in the entire frame, and thusthe same reference sign B is used for denoting the total similaritydegrees. In the present embodiment, the calculation unit 43 calculates,for each l (l=1, 2, . . . , L), a value (composition comparison result)that is obtained by subtracting the square root of the sum of Bd_(l),Bs_(l), Bh_(1l), Bh_(2l) (the distance between the comparative frameF_(j) and the reference frame G₁ in the 4^(th)-order space relating tofour indexes) from 1, and then sets a value obtained by multiplying thisvalue by Bi_(l) (impression comparison result) as the total similaritydegree B₁. Note that a different method may also be employed in otherembodiments, and a value may be obtained, for example, by multiplying asuitable coefficient by the composition comparison result and/or theimpression comparison result, or adding or subtracting a suitable valueto or from the composition comparison result and/or the impressioncomparison result, and then multiplying the results by each other.Alternatively, a value obtained by summing up the square root of1−Bd_(l), the square root of 1−Bs_(l), the square root of 1−Bh_(1l), thesquare root of 1−Bh_(2l), and Bi_(l) while giving a suitable coefficientthereto may be set as the total similarity degree B₁.

With step S47 above, the total similarity degrees B₁, B₂, . . . , B_(L)are calculated. The flow of processing thereafter is the same as in thefirst embodiment.

<4. Usage>

The image processing program 2 can handle image processing with respectto various types of moving images, and can be used also in a scenewhere, for example, an organization such as the police analyzessurveillance images from security cameras in order to investigate anincident. For example, there may be a case where many security camerasare placed in the same shop, and these surveillance images are oftenrecorded in the format of a mixed time-lapse moving image. In such acase, dedicated equipment in the shop in which the security cameras areplaced can be used to separate channels included in the mixed time-lapsemoving image, but in a police facility, which does not always possessthe dedicated equipment that supports devices of manufacturers,reproduction is difficult. Accordingly, the above-described time-lineseparation processing in which a mixed time-lapse moving image can beseparated into moving images of channels regardless of thespecifications of a moving image can be used in such a scene.

<5. Modifications>

Although the embodiments of the present invention have been described sofar, the present invention is not limited to the foregoing embodimentsand various modifications are possible without departing from the spiritof the invention. For example, the following modifications are possible.

<5-1>

The re-sorting window W4 of the foregoing embodiments may be constitutedby a plurality of windows.

<5-2>

In the foregoing embodiments, by switching the selected channel, thedisplay of a frame group in the frame list area C2 can be switched on achannel or sort basis. However, a frame list area for displaying a framegroup in a list may be provided for each of multiple channels and thesort “non-sorted”. In this case, in order to efficiently use a screenspace, thumbnail display for the frame group is preferably performed inthe frame list area, and also a thumbnail image group is preferablysequenced along the time-line. Furthermore, in this case, the channelobjects 91 and the non-sorted object 92 that have an icon format can beomitted, and the frame list area for each channel can be used as achannel object for the association operation, and the frame list areafor the sort “non-sorted” can be used as a non-sorted object for theassociation operation. That is, a configuration is also possible inwhich an arbitrary frame is selected from the “non-sorted” frame listarea, and is dragged and dropped to an arbitrary channel frame list area(channel object), so as to be able to be sorted to the channelcorresponding to the area. The same applies to the case where a frameincluded in a channel is moved to another channel or sort.

Alternatively, a frame list area for displaying the “non-sorted” framegroup in a list and a frame list area for displaying the frame groupbelonging to a channel in a list may be provided, so that the“non-sorted” frame group is constantly displayed in the former area. Inthis case, in the latter area, only the frame group belonging to thecurrently selected channel can be displayed in a list.

<5-3>

In the third embodiment, various color indexes are defined, and arecombined in various manner so as to define the total similarity degreeB₁, but it is also possible to define the color indexes with anothermethod, or combine the color indexes with another method to define thetotal similarity degree B₁. For example, in the third embodiment, acomposition is decided based on the color index regarding density, butthe composition may be decided based on the color index regarding hueand/or chroma. Furthermore, the similarity degree Bs_(i) is calculatedbased on the relative chroma between the sub-regions R₁ and R₂, but adifference in absolute chroma between the sub-regions R₁ and R′₁, or adifference in absolute chroma between the sub-regions R₂ and R′₂ may beset as the similarity degree Bs_(l). Furthermore, a difference indensity between the sub-regions R₁ and R′₁, or a difference in densitybetween the sub-regions R₂ and R′₂ may be calculated as a similaritydegree, and may be added to the total similarity degree B₁.

<5-4>

The foregoing embodiments have a configuration in which the area settingwindow W3 is displayed immediately after the channel separationprocessing has started, so that area setting for preventing erroneousdetermination is possible. However, it is also possible that this areasetting window W3 is not displayed so that such area setting isimpossible. Alternatively, a configuration is also possible in which,although the area setting window W3 is not displayed immediately afterthe channel separation processing has started, the re-sorting window W4has a button for giving a command to restart the automatic sortingprocessing after the area setting has been performed. In this case, onlywhen a result of the automatic sorting processing that was performedwithout performing area setting (the result includes a lot of erroneousdeterminations) is inappropriate, a user presses this button anddisplays the area setting window W3. Then, area setting is performed,and then the automatic sorting processing can again be executed.Alternatively, a configuration is also possible in which the areasetting window W3 is displayed immediately after the channel separationprocessing has started, so that area setting is performed, and theautomatic sorting processing can be executed in this state, andfurthermore, the above-described button is also provided on there-sorting window W4.

<5-5>

In step S9 of the foregoing embodiments, the combination of thecomparative frame F_(j) and the existing reference frame G_(MAX) usingweighted averaging is set as a new reference frame G_(MAX). However, themethod for updating the reference frame G_(MAX) is not limited to this.For example, for each pixel, a histogram of pixel values of all frames(also including the comparative frame F_(j) at this point in time)included in the channel CH_(MAX) may be created, and the mode pixelvalue may be set as the pixel value of this pixel of the reference frameG_(MAX). Alternatively, the pixel value serving as the central value ofthe histogram may be set as the pixel value of this pixel of thereference frame G_(MAX).

<5-6>

In the foregoing embodiments, a configuration can be employed in whichimmediately after step S10 or the like ends, it is determined whether ornot a channel count L has reached a predetermined number, and if it isdetermined that the channel count L has reached the predeterminednumber, processing for correcting an existing channel as with in stepS11 is performed. This is because, if the channel count L increases toomuch, the processing load becomes very large. Note that it is preferableto employ a configuration such that, if there is a channel whose framecount is one, and a temporal distance from this one frame to thecurrently selected comparative frame F_(j) is within a predeterminedrange, the channel is not deleted. This is because this channel is arecently created channel, and a frame to be sorted to this channel islikely to appear in the future. Note that, if even only performing theabove-described channel deletion results in the channel count Lexceeding the predetermined value, it is also possible to sequentiallyset the channels whose count is 2 or more, 3 or more or the like astargets of deletion, and to keep the channel count equal to or smallerthan the predetermined number.

REFERENCE SIGNS LIST

1 Image processing device

2 Image processing program

41 Automatic sorting unit

42 Setting unit

43 Calculation unit

44 Determination unit

45 Re-sorting unit

91 Channel objects

92 Non-sorted object

C3 Play area

W4 Re-sorting window (re-sorting screen)

1. An image processing device for sorting frames included in one movingimage into different channels, the frames included in the moving imagebeing a mixture of frames belonging to the different channels,comprising an automatic sorting unit configured to calculate asimilarity degree of the plurality of frames included in the movingimage by performing image processing on the frames, and sort theplurality of frames into the plurality of channels based on thesimilarity degree; and a re-sorting unit configured to accept, from auser, an association operation of individually associating at least onenon-sorted frame, which is not automatically sorted by the automaticsorting unit, with any of the plurality of channels, and individuallysort the non-sorted frame to any of the plurality of channels based oncontent of the association operation.
 2. The image processing deviceaccording to claim 1, wherein the re-sorting unit is configured todisplay a re-sorting screen for displaying the non-sorted frame, and aplurality of channel objects, which are objects respectivelycorresponding to the plurality of channels, and the associationoperation is an operation for designating, on the re-sorting screen, anyof the plurality of channel objects to the non-sorted frame.
 3. Theimage processing device according to claim 2, wherein each channelobject displays a representative image that represents a frame groupbelonging to a corresponding channel.
 4. The image processing deviceaccording to claim 2, wherein the re-sorting screen is configured todisplay the at least one non-sorted frame in a list.
 5. The imageprocessing device according to claim 4, wherein the re-sorting screenfurther displays a non-sorted object, which is an object correspondingto a sort of putting the at least one non-sorted frame together, and there-sorting unit is configured to accept, from the user, a selectionoperation of selecting one object from among an object group includingthe plurality of channel objects and the non-sorted object, and displaya frame group in a list in a predetermined list display area on there-sorting screen, the frame group belonging to a channel or sort thatcorresponds to the object currently selected by the selection operation.6. The image processing device according to claim 2, wherein there-sorting unit is configured to accept, from the user, a selectionoperation of selecting one object from among an object group includingthe plurality of channel objects, and display a frame group in a list ina predetermined list display area on the re-sorting screen, the framegroup belonging to a channel that corresponds to the channel objectcurrently selected by the selection operation.
 7. The image processingdevice claim 5, wherein, when a frame group belonging to any of theplurality of channels is displayed in a list in the list display area,the re-sorting unit is configured to accept, from the user, an operationfor individually associating a frame that is included in the frame groupdisplayed in the list with any of the plurality of channels, and sortthe frame as per the operation to the channel as per the operation. 8.The image processing device according to claim 2, wherein the re-sortingunit is configured to accept, from the user, a selection operation ofselecting one object from among an object group including the pluralityof channel objects, and reproduce a frame group as a moving image in apredetermined reproduction area on the re-sorting screen, the framegroup belonging to a channel that corresponds to the object currentlyselected by the selection operation.
 9. The image processing deviceaccording to claim 2, wherein the association operation includes anoperation of dragging and dropping the non-sorted frame onto any of theplurality of channel objects, or dragging and dropping any of theplurality of channel objects onto the non-sorted frame, on there-sorting screen.
 10. The image processing device according to claim 2,wherein the re-sorting screen displays a group of frames belonging toeach channel in order of time series in a list.
 11. The image processingdevice according to claim 1, wherein the re-sorting unit is configuredto display information indicating an operation key that corresponds toeach of the plurality of channels, and the association operationincludes operation of the operation key.
 12. The image processing deviceaccording to claim 1, wherein the re-sorting unit is configured torecalculate the frame rate each time the non-sorted frame is sorted toany of the plurality of channels.
 13. The image processing deviceaccording to claim 1, wherein the re-sorting unit is configured toaccept, from the user, an instruction to combine two or more channelsincluded in the plurality of channels, and combine the two or morechannels in response to the instruction for combination.
 14. The imageprocessing device according to claim 1, wherein the re-sorting unit isconfigured to accept, from the user, an instruction to create a channelother than the plurality of channels automatically created by theautomatic sorting unit, and newly create the channel in response to theinstruction for creation.
 15. A non-transitory computer readable mediumstoring an image processing program for sorting frames included in onemoving image into different channels, the frames included in the movingimage being a mixture of frames belonging to the different channels, theimage processing program causing a computer to execute the steps of:calculating a similarity degree of the plurality of frames included inthe moving image by performing image processing on the frames; sortingthe plurality of frames into the plurality of channels based on thesimilarity degree; accepting, from a user, an association operation ofindividually associating at least one non-sorted frame, which is notautomatically sorted, with any of the plurality of channels; andindividually sorting the non-sorted frame to any of the plurality ofchannels based on content of the association operation.
 16. An imageprocessing method for sorting frames included in one moving image intodifferent channels, the frames included in the moving image being amixture of frames belonging to the different channels, the methodcomprising the steps of: calculating a similarity degree of theplurality of frames included in the moving image by performing imageprocessing on the frames; sorting the plurality of frames into theplurality of channels based on the similarity degree; accepting, from auser, an association operation of individually associating at least onenon-sorted frame, which is not automatically sorted, with any of theplurality of channels; and individually sorting the non-sorted frame toany of the plurality of channels based on content of the associationoperation.
 17. The image processing device according to claim 3, whereinthe re-sorting screen is configured to display the at least onenon-sorted frame in a list.
 18. The image processing device according toclaim 3, wherein the re-sorting unit is configured to accept, from theuser, a selection operation of selecting one object from among an objectgroup including the plurality of channel objects, and display a framegroup in a list in a predetermined list display area on the re-sortingscreen, the frame group belonging to a channel that corresponds to thechannel object currently selected by the selection operation.
 19. Theimage processing device according to claim 4, wherein the re-sortingunit is configured to accept, from the user, a selection operation ofselecting one object from among an object group including the pluralityof channel objects, and display a frame group in a list in apredetermined list display area on the re-sorting screen, the framegroup belonging to a channel that corresponds to the channel objectcurrently selected by the selection operation.
 20. The image processingdevice according to claim 6, wherein, when a frame group belonging toany of the plurality of channels is displayed in a list in the listdisplay area, the re-sorting unit is configured to accept, from theuser, an operation for individually associating a frame that is includedin the frame group displayed in the list with any of the plurality ofchannels, and sort the frame as per the operation to the channel as perthe operation.